Computer graphics software and rendering systems provide for the computer synthesis and reproduction of realistic color images and are capable of producing images that include the entire range of visible colors. The broad functions of the typical, computer-based image creation process are illustrated in FIG. 13. As a scene is constructed by the scene designer, an application-specific model captures data about object primitives, and relationships among them, that comprise the scene, in box 20. The scene description, stored in an application database 22, contains descriptions of primitives that define the shapes of components, objects, and light sources in a scene, appearance attributes of the primitives such as color, and surface texture, and connectivity relationships that show how object primitives are positioned and fit together in the scene. The scene description must then be converted into the set of image pixels for display of the image on an output medium, typically on a monitor or in hardcopy form. The process of mapping the scene description to an image pixel file is typically referred to as "rendering", and the implementation of rendering is referred to as "the renderer", shown in FIG. 13 by dotted line box 24.
The specific processes performed in renderer 24 depend on the nature of the image to be produced and on the level of detail captured about the model in the image. The functions shown in boxes 26, 28, and 30 of renderer 24 illustrate a typical renderer for producing realistic images. The application database is traversed in box 26 to select scene description information (objects, attributes, and connectivity information) about the portion of the scene description that is to be viewed, and to convert the information into a format acceptable to the processes in the remainder of the renderer. Model transformation, viewing, and clipping routines, in box 28, scale, rotate, or translate the scene objects, and remove objects, surfaces, and parts of objects and surfaces that should not appear in the displayed image. This process is often referred to as hidden surface removal.
The shaded rendering process, in box 30, takes the transformed, clipped primitives for the objects in the selected view of the scene, simulates the illumination on and in the scene, according to a physically-based model, typically an approximation to the physics of light in the scene, in order to determine a pixel color for every pixel of every primitive in the scene, and produces the image pixels and pixel colors needed to display the scene. Shading attempts to simulate how light in the scene behaves at the surfaces of the objects in the scene. "Ray tracing", generally defined as an image synthesis technique using geometrical optics and rays to recursively evaluate shading and visibility, and "radiosity", generally defined as an image synthesis technique which balances the distribution of light energy in an environment, are two well known shaded rendering methods for image synthesis. Steps in a shaded rendering process typically include first determining which objects (or surfaces of objects) in the scene are seen by the observer from a selected scene viewing position, determining the light arriving at each point on each visible object surface in the scene, simulating the propagation of that light in the scene for that point, and assigning a color value to an image pixel as a function of the light reflected and transmitted at that point by the light sources and other objects in the scene. Once shaded rendering process 30 is completed, the image to be displayed is represented as a collection of individual image pixels, each with an associated pixel color. The image pixels, shown in memory device 32, are typically stored in a frame buffer from which they may be processed, via arrow 34 to box 48, by display routines appropriate for the specific device on which the image is to be generated, for display or reproduction on an output medium 49.
All of the colors physically producible on a particular color output medium is called the color "gamut" of the medium. The color gamut of an existing output medium is typically smaller than the set of all visible colors which are possible in a rendered image. Moreover, the scene description is typically created independently of the output medium on which it is subsequently displayed or reproduced. The algorithms used in shaded rendering process 30 for modeling the light striking a particular point on an object may very well generate a value for the color of one of the object's pixels that is not within the range of the valid values in the color gamut of the output medium, or that numerically represents a non-visible color, such as a negative value. In most instances, a "gamut mapping" process 40 is necessary to make sure that the color attribute for each pixel in the image is displayable or reproducible on the device selected for display or reproduction. Gamut mapping process 40 is described in more detail below, in conjunction with the discussion accompanying FIG. 1.
As can be seen from the functional steps illustrated in FIG. 13, changes that a scene designer may want to make to the image that involve the object primitive level of the image require changing the scene description database 22 and re-rendering the image, which is a computationally expensive procedure. This includes making changes to the colors of objects in an image, since information about objects, including their colors, is only available in the scene description 22. Otherwise, changes to the image made beyond rendering pipeline 24 must be made at the pixel level. Manually changing object or light source colors at the pixel level, while still maintaining the image realism, is difficult for the scene designer to accomplish accurately.
FIG. 1 illustrates a computer-based image creation process which permits a scene designer to manipulate scene description colors, surface properties, and light intensities without the computational expense of re-rendering the entire image. The method of FIG. 1 is described in detail by Carlos H. Sequin and Eliot K. Smyrl in "Parameterized Ray Tracing", Proceedings of SIGGRAPH 1989, Jul. 31-Aug. 4, 1989 Boston Mass., in Computer Graphics 23 (July 1989), pp. 307-314, (hereafter, "Sequin") which is hereby incorporated by reference. The method modifies a conventional ray tracing program of the type which could be used in shaded rendering process 30 of FIG. 13, to produce a parameterized, or symbolic, expression for each pixel as the scene is rendered. Rendering system 25 in FIG. 1 utilizes this modified symbolic shaded rendering process to produce a symbolic pixel expression for each pixel in the rendered image. The symbolic pixel expressions 80 are stored in a memory area. The process of FIG. 1 recognizes that the adjustment of the surface properties and light intensities leaves the scene geometry unchanged. The symbolic pixel expressions 80 explicitly contain, for each pixel in the final image, all potential contributions from the various surfaces hit by the ray and by its reflected and refracted secondary and higher order components, during the ray tracing process. The parameters in these expressions contain symbolic references to the colors and intensities of the various ambient and directional light sources and the coefficients defining the surface properties of the various objects. Color description data 84 contains color values and light intensity values defined by the scene designer for the scene description. Color description data 84, which are derived from the scene description database 22, may be produced by rendering system 25 as separate data, as shown in the separately illustrated file 84 in FIG. 1, or may be incorporated as part of symbolic pixel expression data 80. Each color and light intensity value in color description data 84 can be uniquely identified, such as with a label, index, or pointer, in order to permit the correspondence of an actual color or light intensity value with an associated symbolic reference to a color or intensity in the symbolic pixel expressions. This general indirect referencing arrangement is typically referred to as "indexing", and references hereafter to indexed color description data 84 will include any suitable indirect referencing arrangement for associating the actual color and light intensity values in color description data 84 to the corresponding symbolic references in the symbolic pixel expressions.
The symbolic pixel expression evaluator 82 substitutes the color values and light intensity values stored in color description data 84 for the symbolic color references in the parameters in the symbolic pixel expressions 80 to determine the final color for each pixel, creating the image pixel colors file 32 which represents the final color image. To change the surfaces and lights in a scene, the scene designer changes the color values and light intensity values for the scene description in color description data 84. Symbolic pixel expression evaluator 82 then substitutes the new color values and light intensity values for the symbolic color references in the parameters in the symbolic pixel expressions and recalculates the expressions for new pixel colors in the display. There is thus no need to re-render the scene. As noted in the Sequin paper, parameterization techniques similar to those described in association with ray tracing are applicable to other rendering algorithms which use fixed scene geometries.
The technique of producing symbolic pixel expressions illustrated in FIG. 1, while significantly reducing the time involved in changing colors and light intensities in a scene, may still produce final image pixel colors which are outside the gamut of the output medium which will display or reproduce the image, since the image is created independently of the output medium on which it is subsequently displayed or reproduced. Thus, as shown in FIG. 1, the pixel colors of image pixel file 32 most likely will need to be mapped to the specific color gamut of the output medium, in gamut mapping process 40, before display or reproduction.
Gamut mapping process 40 includes gamut mapping step 42 which applies information about the available color gamut of the output medium from device gamut data file 44 to automated mapping algorithms in order to bring out-of-gamut pixel colors inside the device gamut. Typically, device gamut data file 44 is comprised of data from a sample set of actual colors from the color gamut of the output medium, each of which has been physically measured by an appropriate instrument, such as a colorimeter or spectrophotometer. The output of gamut mapping step 42 is a modified set of image pixel colors, shown in data file 46, that are now in-gamut for a particular output medium. Since each device gamut is different, a new set of image pixel colors is created by the automated gamut mapping process for each output medium on which the image is to be displayed or reproduced.
The goal of gamut mapping is to fit the image colors into the gamut of the output medium while maintaining overall appearance of the image. The gamut mapping process has generally been the concern of those involved in the reproduction or display of color images, and typically has not been a processing goal of the graphics rendering pipeline 24 (FIG. 13) or symbolic shaded rendering system 25 (FIG. 1), nor a primary concern of the scene designer. It is a post-rendering step, accomplished typically by using various automated methods implemented in software which globally or locally modify individual pixel colors without regard to the objects in the image from which the pixels are derived. Gamut mapping may also be accomplished manually. Typically, the gamut of the output device is mathematically represented as a three-dimensional volume in a color space, and pixel colors are presented as points in the color space. One common automated gamut mapping method is a "uniform global scaling" of all of the pixels in the image such that the pixel colors most outside the gamut volume of the device are brought within the gamut region in the color space, and all other pixel colors are scaled in proportion. Uniform global scaling is conceptually similar to "turning down the light" in the scene description. In some, or perhaps many, cases, global scaling of the image pixel colors darkens and desaturates the resulting image unacceptably. Another common automated gamut mapping method involves the "local correction" of pixels, where each pixel is independently examined and brought into the gamut of the specific display device on which the image is to be produced. One undesirable effect of most methods of modifying the out-of-gamut pixels alone, apart from the objects in the scene, is the introduction of shading artifacts into the image.
Another highly undesirable effect of all gamut mapping done at the pixel level is to produce an image which is no longer semantically consistent with the interplay of light with objects in the scene. FIG. 2 of an image 50 illustrates the concept of image semantic inconsistency and highlights the problem with the existing process of modifying individual image pixels having out-of-gamut colors. FIG. 2 illustrates an image 50 of an outdoor street scene composed of a bright red fire truck 52 and an orange construction cone 54, both reflected in a window 55, which reflects only ten percent (10%) of the incident light at the particular viewing angle of the image. In the original synthesized image, suppose that the red color of fire truck 52 is twice as bright as the orange cone and brighter than the brightest displayable red in the gamut of the device on which image 50 is to be displayed, while the orange of construction cone 54 is just within the gamut. The reflections 56 and 58 of the two objects in window 55, generated by rendering pipeline 24 (FIG. 13), are semantically consistent with the original image; that is, the reflection 56 of fire truck 52 is twice as bright as the reflection 58 of the construction cone 54. Because of the darkening effect of the window glass, the red and orange colors of reflections 56 and 58 respectively are within the gamut of the device on which image 50 is to be displayed. The bright red color pixels comprising fire truck 52, generated by the processing in rendering pipeline 24, are undisplayable on the selected output device unless corrected. Uniform global scaling of the image pixel colors will darken both of the objects as well as their reflections, and darkening of the reflections may result in considerable loss of object detail. Local correction of only the out-of-gamut red pixels of fire truck 52 to bring them within the gamut of the device will result in a semantically inconsistent image where the reflections of the object reflect a different relationship between the relative brightness of the objects than what is seen between the objects themselves, once corrected for out-of-gamut colors. This type of semantic inconsistency resulting from a post-rendering gamut mapping process can arise in most rendered scenes which model diffuse and specular inter-reflection of light in the scene.
Automated gamut mapping methods which adjust colors at the pixel level cannot provide image semantic consistency. Thus, in order to achieve both in-gamut pixel colors and semantic consistency of object colors and lights in the image for the device on which the image is to be displayed or reproduced, the scene designer might try to make color changes to the objects in the scene description and re-render the image to try to bring the out-of-gamut colors in certain objects in gamut. If the scene designer is using a symbolic rendering system, such as the one illustrated in FIG. 1, re-rendering the scene would not be necessary, but manual adjustments to the colors in the scene would still be required. While this solution of changing the scene description produces semantically consistent images, it requires the manual adjustment of the scene description or the symbolic pixel expressions, which can be a time consuming process, especially if re-rendering is involved. Alternatively, after the rendering of the image is completed, a user might manually control parameters in automatic mapping algorithms in order to adjust, through trial and error and using artistic judgment, the individual out-of-gamut pixel colors for each of the pixels in the image in order to bring the pixel colors in-gamut in a manner which achieves or almost achieves semantic consistency. This solution, however, rarely provides image semantic consistency because of the complexity of the task, and requires a manual adjustment step which typically involves considerable trial and error because of the relatively unpredictable nature of the changes involved.